Efficient Web Data Mining with Standard XML Technologies

The problem of Web data extraction and XML-based methodology whose goal extends far beyond simple “screen scraping are discussed.” An ideal data extraction process is able to digest target Web databases that are visible only as HTML pages, and create a local, identical replica of those databases as a result. What is needed in this process is much more than a Web crawler and set of Web site wrappers. A comprehensive data extraction process needs to deal with such roadblocks such as session identifiers, HTML forms, and client-side JavaScript, and data integration problems such as incompatible datasets and vocabularies, and missing and conflicting data. Proper data extraction also requires a solid data validation and error recovery service to handle data extraction failures, which are unavoidable. In this paper we describe NDES, a software framework that makes significant advances in solving these problems and provides a platform for building a productionquality Web data extraction process. Key aspects of NDES are that it uses XML technologies for data extraction, including XHTML and XSLT, and provides access to the “deep Web.”